XG Boost in Mental Health Classification

Introduction

eXtreme Gradient Boosting (XGB) algorithm

• A supervised machine learning algorithm
• Modification of Gradient Boosting Framework
• Ensemble of weak decision trees
• L1,L2 regularization
• High performance, speed, scalability
• Applications in healthcare, education, public health, finance, and engineering.

Introduction

XGBoost for mental disorder classification

• Using XGBoost on clinical and survey data
• Classification of individuals into one of four categories: Bipolar I Disorder, Bipolar II Disorder, Major Depressive Disorder, and Normal
• XGBoosts proves to be a suitable algorithm for the problem

Literature Review

eXtreme Gradient Boosting (XGBoost) in healthcare

• XGBoost strengths in real world problems handling class imbalance, heterogeneous data types, or non-linear relationships. (Chen and Guestrin 2016)
• XGBoost combined with DL for breast cancer classification with high reliability. (Liew, Hameed, and Clos 2021)
• XGBoost with biomarker data to improve depression diagnoses in a large Dutch population dataset.(Sharma and Verbeke 2020)
• XGBoost to multi-modal datasets to predict self-harm in young adults.(Xu et al. 2024)
• A hybrid algorithm (XGBoost-HOA) to classify depression, anxiety, and stress
• Dual XGBoost models applied to distinguish between deficit and non-deficit schizophrenia subtypes using fMRI features.(Zhang, Jia, and Shang 2022)
• XGBoost compared with linear regression in predicting depression among refugee children.(Saleh et al. 2024)

Literature Review

XGBoost limitations for imbalance data

• handled by optimization techniques (Zhang, Jia, and Shang 2022)
• Handled by balancing the class distribution (Sharma and Verbeke 2020)
• Preprocessing and hyperparameter tuning importance in imbalance data for accurate XGBoost algorithm.

Literature Review

XGBoost in Education

• XGBoost in academic prediction models (Hu and Song 2019)
• XGBoost out performing traditional logistic regression models in predicting learner performance (Hakkal and Ait Lahcen 2024).

XGBoost in Public health

• XGBoost used to predict daily COVID-19 cases in the United States.(Fang et al. 2022)
• A hybrid model integrating XGBoost, Random Forest, and Antlion Optimization used to predict infectious disease outbreaks.(Sivakumar and Elangovan 2023)

Literature Review

XGBoost in financial sector

• XGBoost model’s ability to rank feature importance and prevent overfitting using regularization made it a top performer in credit risk prediction tasks(Li et al. 2020)
• XGBoost to forecast volatility in the U.S. stock market, identifying the Economic Policy Uncertainty Index as a critical predictor.(Fomunyam 2023)

XGBoost In pharmaceutical research

• Application of XGBoost in drug development.(Wiens et al. 2025)

Literature Review

XGBoost in educational diagnostics

• XGBoost enhanced learner performance prediction and knowledge tracing on platforms like ASSIST09 and Algebra08. (Su et al. 2023) and (Hakkal and Ait Lahcen 2024)

XGBoost in Sports Analytics

• XGBoost used to predict ultramarathon running speeds based on demographics and environmental factors.(Nikolaidis, Knechtle, and co-authors 2023)

Methods: Extreme Gradient Boosting (XGBoost)

• Ensemble of weak decision trees
• Trained sequentially, each tree corrects previous errors
• Captures non-linear relationships within the 17-variable dataset
• Uses gradient descent to minimize loss
• Includes regularization to prevent overfitting

Methods: XGBoost Formulas

• Loss term: Measures prediction error
• Regularization term: Penalizes model complexity
• Ensures balance between accuracy and generalization

\[ \text{Obj} = \sum_{i=1}^{n} L(y_i, \hat{y}_i) + \sum_{k=1}^{K} \Omega(f_k) \]

Data Exploration and Visualization

• Sourced from Kaggle (2023 clinical records)
• 120 individuals assessed for mental disorders -Labels: -Normal -Bipolar Type I -Bipolar Type II -Depression
• 17 diagnostic variables
• Mix of clinical, behavioral, demographic factors

Data Exploration and Visualization

• Examined class distribution
• Identified potential imbalance or outliers
• Explored correlations between features

Modeling and Results

Data Preprocessing

To facilitate analysis, all categorical variables in the dataset were transformed into numerical or grouped categorical formats in Excel. The transformations were applied as follows:

• Ordinal responses (Sadness, Euphoric, Exhausted) recoded:
  • Seldom=1, Sometimes=2, Usually=3, Most-Often=4
• Binary items (Suicidal Thoughts, Mood Swing):
  • Yes=1, No=0
• 1–10 rating scales (Sexual Activity, Optimism) grouped:
  • 1–3 → Cat 1, 4–6 → Cat 2, 7–9 → Cat 3

Modeling and Results

Hyperparameter Optimization

• The dataset was randomly partitioned into an 80/20 training–testing split.
  • This split resulted in 96 patient surveys used for model training and 24 surveys reserved for independent testing.
  • The predictive target for all models was the expert clinical diagnosis
• An XGBoost model specification was created using the tidymodels framework where all major hyperparameters were set to be tuned including: - the number of trees - tree depth - learning rate - loss reduction, - minimum node size - subsampling rate - number of predictors considered at each split

Modeling and Results

Hyperparameter Optimization

• Five-fold stratified cross-validation was used for hyperparameter tuning.
  • Where the training dataset was partitioned into five equally sized folds
  • For each tuning iteration:
    • Models were trained on four folds
    • Validated on the remaining fold
    • Which ensured stable and unbiased performance estimates

(singh2024crossvalidation_image?)

Modeling and Results

Hyperparameter Optimization

• The final hyperparameter grid consists of 30 candidate hyperparameter combination which was constructed using a space-filling design.
  • Model performance was assessed using accuracy and the area under the ROC curve (ROC AUC).
  • Visualizing the tuning results revealed the ROC AUC versus number of trees plot indicated that the optimal tree depth was approximately 2, suggesting that relatively shallow trees generalized best for this dataset.
  • Model performance increased steadily with the number of boosting iterations, with the highest ROC AUC occurring at around 625 trees. At this configuration, the model achieved an accuracy exceeding 0.80 and a ROC AUC greater than 0.90, indicating strong discriminatory performance.

VisulizationTuningResults

Modeling and Results

Hyperparameter Optimization

• The select_best() function from the tidymodels framework was used to systematically extract the hyperparameter combination that achieved the highest ROC AUC across all resampled evaluations.

• The resulting optimal configuration included 3 predictors (mtry = 3), 627 boosting iterations, a minimum node size (min_n) of 2, a learning rate of 0.16501, a loss-reduction parameter of 0.0034, and a subsample proportion of 0.5655.

  • This combination represents the tuning grid entry labeled pre0_mod06_post0, which achieved the best balance between model complexity and predictive performance during cross-validation.

BestParams

Modeling and Results

Model Evaluation & Performance

Prediction

The model’s predictions on the held-out test set showed strong agreement with the expert diagnoses. Specifically:

• Bipolar Type-1: 4 of 6 cases were correctly predicted (66.7%)
  
• Bipolar Type-2: 5 of 7 cases were correctly predicted (71.4%)
  
• Depression: 6 of 7 cases were correctly predicted (85.7%)
  
• Normal: 4 of 6 cases were correctly predicted (66.7%)

Modeling and Results

Model Evaluation & Performance

The XGBoost multiclass classification model demonstrates moderate overall predictive performance, with room for improvement but also several encouraging indicators of discriminative ability. - Accuracy: 0.654
An accuracy of 65.4% suggests that the model correctly classifies roughly two-thirds of the observations. - Kappa: 0.534
The Kappa value of 0.534 reflects moderate agreement between the model’s predictions and the true class labels after accounting for chance. - ROC AUC (macro-weighted): 0.927
The macro-weighted ROC AUC of 0.927 suggests that the model does a strong job of ranking observations correctly across all classes, even if it occasionally makes incorrect final class assignments.

Taken together, the metrics indicate that the model has strong underlying discriminative power (high AUC) but moderate final classification performance (accuracy and kappa).

Modeling and Results

Feature Importance & Interepretability

• A SHAP values analysis was performed to highlight how predictors influence classification across four classes, revealing both broadly influential and class-specific features.

• Most Influential Features (Across Classes)

  • Mood Swing – Strongest overall predictor, especially for Bipolar Type-1 and Type-2.
    
  • Suicidal thoughts – Highly impactful for Depression and Bipolar Type-2.
    
  • Sadness – Major differentiator for Bipolar Type-1.

• Class-Specific Influences

  • Concentration Category 2 – Strong for Normal
  • Authority & Respect – Most impactful for Bipolar Type-2.
    
  • Ignore & Move On & Exhausted – Distinguish the Normal class

• Overall Conclusion

  • Distinct SHAP patterns across classes indicate unique psychological signatures, consistent with a well-calibrated multiclass XGBoost model.

Modeling and Results

Sensitvity & Robustness Analysis

• The sensitivity analysis results indicate that the model performs consistently and reliably across repeated resampling or perturbation scenarios. The metrics summarize how stable the model’s performance remains when exposed to variability in the input data.
  • Mean Accuracy: 0.804 reflects strong overall predictive performance across iterations.
  • SD of Accuracy: 0.0821 suggests moderate variability, meaning that although the model generally performs well, accuracy fluctuates somewhat depending on sample composition.
  • Mean ROC AUC: 0.957 demonstrates excellent discriminative ability, indicating that the model consistently separates classes with high precision.
    
  • SD of ROC AUC: 0.0397 shows that this discriminative performance remains highly stable, even when the data varies. These results show that:
  • The model has strong overall performance, especially in terms of ROC AUC.
    
  • Variability across repeats is low to moderate, supporting the model’s robustness.
    
  • The model’s ability to rank or distinguish classes (ROC AUC) is especially stable, while accuracy shows slightly more sensitivity to data shifts.

Modeling and Results

Summary of Modeling Results

• On the independent 24-case test set, the model showed strong agreement with expert diagnoses
• The model successfully identified the majority of cases in each diagnostic category, demonstrating strong generalization to unseen data.

The final XGBoost model delivered: - High discriminative performance, with ROC AUC > 0.90 across cross-validation and sensitivity analyses
- Stable, robust generalization, supported by low performance variability
- Interpretable, clinically relevant insights, enabled by SHAP and feature importance
- Accurate classification, with strong performance across all four diagnostic groups on unseen data

Overall, XGBoost proved to be an effective and interpretable modeling approach for mental health classification in a small structured dataset, delivering both predictive value and meaningful interpretability essential for clinical research.

Conclusion: Study Purpose

• Evaluate effectiveness of XGBoost for predicting mental health outcomes.
• Develop, tune, and validate the model.
• Identify highest‑predictive features and assess ability to capture non‑linear relationships.
• Model showed strong accuracy and generalizability.

Conclusion: Key Findings

• Top predictors: stress indicators, behavioral patterns, wellness factors.
• High feature importance suggests relevance to mental‑health risk.
• Model effectively distinguished between classes.
• Stable performance across tuning configurations with minimal overfitting.

Conclusion: Implications

• XGBoost shows potential for early warning mental‑health monitoring.
• Useful for identifying at‑risk individuals based on survey data.
• Self‑reported data may introduce bias—interpret with care.

Conclusion: Future Work

• Use larger and more diverse datasets.
• Incorporate behavioral or time‑based variables.
• Compare with Random Forest, LightGBM, neural networks.
• Test in real‑world settings.
• Address fairness and ethical use of health data.

Conclusion: Overview

• XGBoost is effective for survey‑based mental‑health prediction.
• Provides fast and interpretable insights alongside traditional assessments.
• Predictive modeling could expand tools for mental‑health research and practice.